Infinispan User Guide

However, users do not have to first read an XML based configuration and then modify it in runtime; they can start from scratch using only programmatic API. This is where powerful ConfigurationBuilder API comes to shine. The aim of this API is to make it easier to chain coding of configuration options in order to speed up the coding itself and make the configuration more readable. This new configuration can be used for both the global and the cache level configuration. GlobalConfiguration objects are constructed using GlobalConfigurationBuilder while Configuration objects are built using ConfigurationBuilder. Let’s look at some examples on configuring both global and cache level options with this new API:

One of the most commonly configured global option is the transport layer, where you indicate how an Infinispan node will discover the others:

Sometimes you might also want to get global JMX statistics and information about the transport, or the cache manager in general. To enable global JMX statistics simply do:

Further options at the global JMX statistics level allows you for example to configure the cache manager name which comes handy when you have multiple cache managers running on the same system, or how to locate the JMX MBean Server:

Some of the Infinispan features are powered by a group of the thread pool executors which can also be tweaked at this global level. For example:

You can not only configure global, cache manager level, options, but you can also configure cache level options such as the cluster mode :

Or you can configure eviction/expiration settings to:

An application might also want to interact with an Infinispan cache within the boundaries of JTA and to do that you need to configure the transaction layer and optionally tweak the locking settings . When interacting with transactional caches, you might want to enable recovery to deal with transactions that finished with an heuristic outcome and if you do that, you will often want to enable JMX management and statistics gathering too:

Configuring Infinispan with chained cache stores is simple too:

The fluent configuration can also be used to configure more advanced or exotic options, such as advanced externalizers :

Or, add custom interceptors :

For information on the individual configuration options, please check the configuration guide .

If you are configuring your cache programmatically, all you need to do is:

and if you happen to use an XML file to configure Infinispan, just use:

In both cases above, Infinispan looks for jgroups.xml first in your classpath, and then for an absolute path name if not found in the classpath.

Infinispan ships with a few different JGroups files (packaged in infinispan-core.jar) which means they will already be on your classpath by default. All you need to do is specify the file name, e.g., instead of jgroups.xml above, specify jgroups-tcp.xml.

The configurations available are:

JGroups also supports more system property overrides, details of which can be found on this page: SystemProps

In addition, the JGroups configuration files shipped with Infinispan are intended as a jumping off point to getting something up and running, and working. More often than not though, you will want to fine-tune your JGroups stack further to extract every ounce of performance from your network equipment. For this, your next stop should be the JGroups manual which has a detailed section on configuring each of the protocols you see in a JGroups configuration file.

Hot Rod client does not support dynamically start/stop of cache. == The Cache APIs

Certain methods exposed in Map have certain limitations when used with Infinispan, such as size() , values() , keySet() and entrySet() . Specifically, these methods are unreliable and only provide a best-effort guess. They do not acquire locks, either local or global, and concurrent modifications, additions and removals will not be considered in the result of any of these calls. Further, they only operate on the local node, and as such, do not give you a global(cluster) view of the state.

Attempting to perform these operations globally would have large performance impact as well as become a scalability bottleneck. As such, these methods should only be used for informational or debugging purposes only.

Further to simply storing entries, Infinispan’s cache API allows you to attach mortality information to data. For example, simply using put(key, value) would create an immortal entry, i.e., an entry that lives in the cache forever, until it is removed (or evicted from memory to prevent running out of memory). If, however, you put data in the cache using put(key, value, lifespan, timeunit) , this creates a mortal entry, i.e., an entry that has a fixed lifespan and expires after that lifespan.

In addition to lifespan , Infinispan also supports maxIdle as an additional metric with which to determine expiration. Any combination of lifespans or maxIdles can be used.

See this page for an example of using mortal data with Infinispan

Infinispan’s Cache class contains a different put operation called putForExternalRead . This operation is particularly useful when Infinispan is used as a temporary cache for data that is persisted elsewhere. Under heavy read scenarios, contention in the cache should not delay the real transactions at hand, since caching should just be an optimization and not something that gets in the way.

To achieve this, putForExternalRead acts as a put call that only operates if the key is not present in the cache, and fails fast and silently if another thread is trying to store the same key at the same time. In this particular scenarion, caching data is a way to optimise the system and it’s not desirable that a failure in caching affects the on-going transaction, hence why failure is handled differently. putForExternalRead is consider to be a fast operation because regardless of whether it’s successful or not, it doesn’t wait for any locks, and so returns to the caller promptly.

To understand how to use this operation, let’s look at basic example. Imagine a cache of Person instances, each keyed by a PersonId , whose data originates in a separate data store. The following code shows the most common pattern of using putForExternalRead within the context of this example:

Please note that putForExternalRead should never be used as a mechanism to update the cache with a new Person instance originating from application execution (i.e. from a transaction that modifies a Person’s address). When updating cached values, please use the standard put operation, otherwise the possibility of caching corrupt data is likely.

Flags are applied to regular cache methods to alter the behavior of certain methods. For a list of all available flags, and their effects, see the Flag enumeration. Flags are applied using AdvancedCache.withFlags() . This builder method can be used to apply any number of flags to a cache invocation, for example:

The AdvancedCache interface also offers advanced developers a mechanism with which to attach custom interceptors. Custom interceptors allow developers to alter the behavior of the cache API methods, and the AdvancedCache interface allows developers to attach these interceptors programmatically, at run-time. See the AdvancedCache Javadocs for more details.

For more information on writing custom interceptors, see /javascript:;

Cache-level events occur on a per-cache basis, and is global and cluster-wide. Examples of cache-level events are entries being added, removed, modified, etc. These events trigger notifications to listeners registered to a specific cache.

Please see the Javadocs on the org.infinispan.notifications.cachelistener.annotation package for a comprehensive list of all cache-level notifications, and their respective method-level annotations.

Cache manager-level events occur on a cache manager. These too are global and cluster-wide, but involve events that affect all caches created by a single cache manager. Examples of cache manager-level events are nodes joining or leaving a cluster, or caches starting or stopping.

Please see the Javadocs on the org.infinispan.notifications.cachemanagerlistener.annotation package for a comprehensive list of all cache manager-level notifications, and their respective method-level annotations.

By default, all notifications are dispatched in the same thread that generates the event. This means that you must write your listener such that it does not block or do anything that takes too long, as it would prevent the thread from progressing. Alternatively, you could annotate your listener as asynchronous , in which case a separate thread pool will be used to dispatch the notification and prevent blocking the event originating thread. To do this, simply annotate your listener such:

At the moment there is no support for registering listeners for the RemoteCacheManager . Whenever calling one of the add/remove/getListener methods on the RemoteCacheManager an UnsupportedOperationException will be thrown. There are plans to support remote listeners in future versions; in order to be notified when this feature will be added you can watch/vote for ISPN-374 .

Non-blocking APIs are powerful in that they provide all of the guarantees of synchronous communications - with the ability to handle communication failures and exceptions - with the ease of not having to block until a call completes.  This allows you to better harness parallelism in your system.  For example:

There are 4 things in Infinispan that can be considered to be on the critical path of a typical write operation. These are, in order of cost:

As of Infinispan 4.0, using the async methods will take the network calls and marshalling off the critical path.  For various technical reasons, writing to a cache store and acquiring locks, however, still happens in the caller’s thread.  In future, we plan to take these offline as well.  See this developer mail list thread about this topic.

Strictly, these methods do not return JDK Futures, but rather a sub-interface known as a NotifyingFuture .  The main difference is that you can attach a listener to a NotifyingFuture such that you could be notified when the future completes.  Here is an example of making use of a notifying future:

The Javadocs on the Cache interface has some examples on using the asynchronous API, as does this article by Manik Surtani introducing the API.

Another approach would be to use the DecoratedCache wrapper. This allows you to reuse flags. For example:

This approach makes your code more readable.

If you want to use these or any other flags available, which by the way are described in detail the Flag enumeration , you simply need to get hold of the advanced cache and add the flags you need via the withFlags() method call. For example:

It’s worth noting that these flags are only active for the duration of the cache operation. If the same flags need to be used in several invocations, even if they’re in the same transaction, withFlags() needs to be called repeatedly. Clearly, if the cache operation is to be replicated in another node, the flags are carried over to the remote nodes as well.

Infinispan allocates each node a portion of the total hash space. Normally, when you store an entry, Infinispan will take a hash of the key, and store the entry on the node which owns that portion of the hash space. Infinispan always uses an algorithm to locate a key in the hash space, never allowing the node on which the entry is stored to be specified manually. This scheme allows any node to know which nodes owns a key, without having to distribute such ownership information. This reduces the overhead of Infinispan, but more importantly improves redundancy as there is no need to replicate the ownership information in case of node failure.

If you use the grouping API , then Infinispan will ignore the hash of the key when deciding which node to store the entry on, and instead use a hash of the group. Infinispan still uses the hash of the key to store the entry on a node. When the group API is in use, it is important that every node can still compute, using an algorithm, the owner of every key. For this reason, the group cannot be specified manually. The group can either be intrinsic to the entry (generated by the key class) or extrinsic (generated by an external function).

First, you must enable groups. If you are configuring Infinispan programmatically, then call:

Or, if you are using XML:

If you have control of the key class (you can alter the class definition, it’s not part of an unmodifiable library), and the determination of the group is not an orthogonal concern to the key class, then we recommend you use an intrinsic group. The intrinsic group can be specified using the @Group annotation placed on the method. Let’s take a look at an example:

If you don’t have control over the key class, or the determination of the group is an orthogonal concern to the key class, we recommend you use an extrinsic group. In extrinsic group is specified by implementing the Grouper interface, which has a single method computeGroup, which should return the group. Grouper acts as an interceptor, passing the previously computed value in. The group passed to the first Grouper will be that determined by @Group (if @Groupis defined). This allows you even greater control over the group when using an intrinsic group. For a grouper to be used when determining the group for a key, it’s keyType must be assignable from the key being grouped.

Let’s take a look at an example of a Grouper:

Here we determine a simple grouper that can take the key class and extract from the group from the key using a pattern. We ignore any group information specified on the key class.

You must register every grouper you wish to have used. If you are configuring Infinispan programmatically:

Or, if you are using XML:

The aim of this API is to store information in a hierarchical way. The hierarchy is defined using paths represented as Fqn or fully qualified names , for example: /this/is/a/fqn/path or /another/path . In the hierarchy, there’s an special path called root which represents the starting point of all paths and it’s represented as: /

Each FQN path is represented as a node where users can store data using a key/value pair style API (i.e. a Map). For example, in /persons/john , you could store information belonging to John, for example: surname=Smith, birthdate=05/02/1980…etc.

Please remember that users should not use root as a place to store data. Instead, users should define their own paths and store data there. The following sections will delve into the practical aspects of this API.

The first step to use the tree API is to actually create a tree cache. To do so, you need to create an Infinispan Cache as you’d normally do , and using the TreeCacheFactory , create an instance of TreeCache . A very important note to remember here is that the Cache instance passed to the factory must be configured with invocation batching . For example:

The Tree API effectively provides two ways to interact with the data:

Via TreeCache convenience methods: These methods are located within the TreeCache interface and enable users to store , retrieve , move , remove …etc data with a single call that takes the Fqn , in String or Fqn format, and the data involved in the call. For example:

Or:

Via Node API: It allows finer control over the individual nodes that form the FQN, allowing manipulation of nodes relative to a particular node. For example:

Or:

Or even:

A node also provides the ability to access its parent or children . For example:

Or:

In the previous section, some of the most used operations, such as addition and retrieval, have been shown. However, there are other important operations that are worth mentioning, such as remove:

You can for example remove an entire node, i.e. /persons/john , using:

Or remove a child node, i.e. persons that a child of root, via:

You can also remove a particular key/value pair in a node:

Or you can remove all data in a node with:

Another important operation supported by Tree API is the ability to move nodes around in the tree. Imagine we have a node called "john" which is located under root node. The following example is going to show how to we can move "john" node to be under "persons" node:

Current tree structure:

Moving trees from one FQN to another:

Final tree structure:

Understanding when and how locks are acquired when manipulating the tree structure is important in order to maximise the performance of any client application interacting against the tree, while at the same time maintaining consistency.

Locking on the tree API happens on a per node basis. So, if you’re putting or updating a key/value under a particular node, a write lock is acquired for that node. In such case, no write locks are acquired for parent node of the node being modified, and no locks are acquired for children nodes.

If you’re adding or removing a node, the parent is not locked for writing. In JBoss Cache, this behaviour was configurable with the default being that parent was not locked for insertion or removal.

Finally, when a node is moved, the node that’s been moved and any of its children are locked, but also the target node and the new location of the moved node and its children. To understand this better, let’s look at an example:

Imagine you have a hierarchy like this and we want to move c/ to be underneath b/:

The end result would be something like this:

To make this move, locks would have been acquired on:

The current Infinispan listeners have been designed with key/value store notifications in mind, and hence they do not map to tree cache events correctly. Tree cache specific listeners that map directly to tree cache events (i.e. adding a child…etc) are desirable but these are not yet available. If you’re interested in this type of listeners, please follow this issue to find out about any progress in this area.

This eviction strategy effectively disables the eviction thread.

UNORDERED eviction strategy is a legacy eviction strategy that has been deprecated. If UNORDERED strategy is specified LRU eviction algorithm will be used.

If LRU eviction is used cache entries are selected for eviction using a well known least-recently-used pattern.

LRU eviction algorithm, although simple and easy to understand, under performs in cases of weak access locality (one time access entries are not timely replaced, entries to be accessed soonest are unfortunately replaced, and so on). Recently, a new eviction algorithm - LIRS has gathered a lot of attention because it addresses weak access locality shortcomings of LRU yet it retains LRU’s simplicity. Eviction in LIRS algorithm relies on history information of cache entries accesses using so called Inter-Reference Recency (a.k.a IRR) and the Recency. The IRR of a cache entry A refers to number of other distinct entries accessed between the last two consecutive accesses to cache entry A, while recency refers to the number of other entries accessed from last reference to A up to current time point. If we relied only on cache recency we would essentially have LRU functionality. However, in addition to recency LIRS tracks elements that are in low IRR and high IRR, aptly named LIR and HIR cache entry blocks respectively. LIRS eviction algorithm essentially keeps entries with a low IRR in the cache as much as possible while evicting high IRR entries if eviction is required. If recency of a LIR cache entry increases to a certain point and entry in HIR gets accessed at a smaller recency than that of the LIR entry, the LIR/HIR statuses of the two blocks are switched. Entries in HIR may be evicted regardless of its recency, even if element was recently accessed.

By default when no <eviction /> element is specified, no eviction takes place.

In case there is an eviction element, this table describes behaviour of eviction based on information provided in the xml configuration ("-" in Supplied maxEntries or Supplied strategy column means that the attribute wasn’t supplied)

Implementing eviction in a scalable, low lock contention approach while at the same time doing meaningful selection of entries for eviction is not an easy feat. Data container needs to be locked until appropriate eviction entries are selected. Having such a lock protected data container in turn causes high lock contention offsetting any eviction precision gained by sophisticated eviction algorithms. In order to get superior throughput while retaining high eviction precision both low lock contention data container and high precision eviction algorithm implementation are needed. Infinispan evicts entries from cache on a segment level (segments similar to ConcurrentHashMap), once segment is full entries are evicted according to eviction algorithm. However, there are two drawbacks with this approach. Entries might get evicted from cache even though maxEntries has not been reached yet and maxEntries is a theoretical limit for cache size but in practical terms it will be slightly less than maxEntries. For more details refer to Infinispan eviction design.

Both Eviction and Expiration are means of cleaning the cache of unused entries and thus guarding the heap against OutOfMemory exceptions, so now a brief explanation of the difference.

With eviction you set maximal number of entries you want to keep in the cache and if this limit is exceeded, some candidates are found to be removed according to a choosen eviction strategy (LRU, LIRS, etc…). Eviction can be setup to work with passivation (evicting to a cache store).

With expiration you set time criteria for entries, how long you want to keep them in cache. Either you set maximum lifespan of the entry - time it is allowed to stay in the cache or maximum idle time , time it’s allowed to be untouched (no operation performed with given key).

Eviction may be configured using the Configuration bean or the XML file. Eviction configuration is on a per-cache basis. Valid eviction-related configuration elements are:

Programmatically, the same would be defined using:

Eviction is disabled by default. If enabled (using an empty <eviction /> element), certain default values are used:

Expiration lifespan and maxIdle both default to -1.

Expiration allows you to set either a lifespan or a maximum idle time on each key/value pair stored in the cache. This can either be set cache-wide using the configuration, as described above, or it can be defined per-key/value pair using the Cache interface. Any values defined per key/value pair overrides the cache-wide default for the specific entry in question.

For example, assume the following configuration:

With optimistic transactions locks are being acquired at transaction prepare time and are only being held up to the point the transaction commits (or rollbacks). This is different from the 5.0 default locking model where local locks are being acquire on writes and cluster locks are being acquired during prepare time.

Optimistic transactions can be enabled in the configuration file:

or programatically:

From a lock acquisition perspective, pessimistic transactions obtain locks on keys at the time the key is written. E.g.

When cache.put(k1,v1) returns, k1 is locked and no other transaction running anywhere in the cluster can write to it. Reading k1 is still possible. The lock on k1 is released when the transaction completes (commits or rollbacks).

Pessimistic transactions can be enabled in the configuration file:

or programatically:

The autoCommit attribute was added in order to assure backward compatibility with Infinispan 4. If a cache is transactional and autoCommit is enabled (defaults to true) then any call that is performed outside of a transaction’s scope is transparently wrapped within a transaction. In other words Infinispan adds the logic for starting a transaction before the call and committing it after the call.

Therefore if your code accesses a cache both transactionally and non-transactionally all you have to do when migrating to Infinispan 5.1 is mark the cache as transactional and enable autoCommit (that’s actually enabled by default)

The autoCommit feature can be managed through configuration:

From a use case perspective, optimistic transactions should be used when there is not a lot of contention between multiple transactions running at the same time. That is because the optimistic transactions rollback if data has changed between the time it was read and the time it was committed (writeSkewCheck).

On the other hand, pessimistic transactions might be a better fit when there is high contention on the keys and transaction rollbacks are less desirable. Pessimistic transactions are more costly by their nature: each write operation potentially involves a RPC for lock acquisition.

Once you have configured your cache to use batching, you use it by calling startBatch() and endBatch() on Cache. E.g.,

Behind the scenes, the batching functionality starts a JTA transaction, and all the invocations in that scope are associated with it. For this it uses a very simple (e.g. no recovery) internal TransactionManager implementation. With batching, you get:

Consider a distributed transaction in which money is transferred from an account stored in an external database to an account stored in Infinispan. When TransactionManager.commit() is invoked, both resources prepare successfully (1st phase). During the commit (2nd) phase, the database successfully applies the changes whilst Infinispan fails before receiving the commit request from the transaction manager. At this point the system is in an inconsistent state: money is taken from the account in the external database but not visible yet in Infinispan (since locks are only released during 2nd phase of a two-phase commit protocol). Recovery deals with this situation to make sure data in both the database and Infinispan ends up in a consistent state.

Recovery is coordinated by the transaction manager. The transaction manager works with Infinispan to determine the list of in-doubt transactions that require manual intervention and informs the system administrator (via email, log alerts, etc). This process is transaction manager specific, but generally requires some configuration on the transaction manager.  

Knowing the in-doubt transaction ids, the system administrator can now connect to the Infinispan cluster and replay the commit of transactions or force the rollback. Infinispan provides JMX tooling for this - this is explained extensively in the Reconciliate state section.

Recovery is not enabled by default in Infinispan. If disabled the TransactionManager won’t be able to work with Infinispan to determine the in-doubt transactions. In order to enable recovery through xml configuration:

NOTE:  the recoveryInfoCacheName attribute is not mandatory.

Alternatively you can enable it through the fluent configuration API as follows:

Recovery can be enabled/disabled o a per cache level: e..g it is possible to have a transaction spanning a cache that is has it enabled and another one that doesn’t.

In order to track in-doubt transactions and be able to reply them Infinispan caches all transaction state for future use. This state is held only for in-doubt transaction, being removed for successfully completed transactions after the commit/rollback phase completed.

This in-doubt transaction data is held within a local cache: this allows one to configure swapping this info to disk through cache loader in the case it gets too big. This cache can be specified through the  "recoveryInfoCacheName" configuration attribute. If not specified infinispan will configure a local cache for you.

It is possible (though not mandated) to share same recovery cache between all the Infinispan caches that have recovery enabled.  If default recovery cache is overridden then the specified recovery cache must use a TransactionManagerLookup that returns a different transaction manager than the one used by the cache itself.

Even though this is transaction manager specific, generally a transaction manager would need a reference to an XAResource implementation in order to invoke XAResource.recover() on it. In order to obtain a reference to an Infinispan XAResource following API can be used:

It is a common practice to run the recovery in a different process from the one running the transaction. At the moment it is not possible to do this with infinispan: the recovery must be run from the same process where the infinispan instance exists. This limitation will be dropped once transactions over HotRod are available.

The transaction manager informs the system administrator on in-doubt transaction in a proprietary way. At this stage it is assumed that the system administrator knows transaction’s XID (a byte array).

A normal recovery flow is:

image::[]

The status of each in-doubt transaction is displayed(in this example " PREPARED "). There might be multiple elements in the status field, e.g. "PREPARED" and "COMMITTED" in the case the transaction committed on certain nodes but not on all of them.  

image::[]

The recovery design document describes in more detail the insides of transaction recovery implementation.

Infinispan offers two isolation levels - READ_COMMITTED (the default) and REPEATABLE_READ, configurable via the <locking /> configuration element.

These isolation levels determine when readers see a concurrent write, and are internally implemented using different subclasses of MVCCEntry, which have different behaviour in how state is committed back to the data container.

Here’s a more detailed example that should help understand the difference between READ_COMMITTED and REPEATABLE_READ in the context of Infinispan. With read committed, if between two consecutive read calls on the same key, the key has been updated by another transaction, the second read will return the new updated value:

With REPEATABLE_READ, step 7 will still return v. So, if you’re going to retrieve the same key multiple times within a transaction, you should use REPEATABLE_READ.

The LockManager is a component that is responsible for locking an entry for writing. The LockManager makes use of a LockContainer to locate/hold/create locks. LockContainers come in two broad flavours, with support for lock striping and with support for one lock per entry.

Lock striping entails the use of a fixed-size, shared collection of locks for the entire cache, with locks being allocated to entries based on the entry’s key’s hash code. Similar to the way the JDK’s ConcurrentHashMap allocates locks, this allows for a highly scalable, fixed-overhead locking mechanism in exchange for potentially unrelated entries being blocked by the same lock.

The alternative is to disable lock striping - which would mean a new lock is created per entry. This approach may give you greater concurrent throughput, but it will be at the cost of additional memory usage, garbage collection churn, etc.

The size of the shared lock collection used by lock striping can be tuned using the concurrencyLevel attribute of the <locking /> configuration element.

In addition to determining the size of the striped lock container, this concurrency level is also used to tune any JDK ConcurrentHashMap based collections where related, such as internal to DataContainers. Please refer to the JDK ConcurrentHashMap Javadocs for a detailed discussion of concurrency levels, as this parameter is used in exactly the same way in Infinispan.

The fact that a single owner is locked (as opposed to all owners being locked) does not break the following consistency guarantee: if key K is hashed to nodes {A, B} and transaction TX1 acquires a lock for K, let’s say on A. If another transaction, TX2, is started on B (or any other node) and TX2 tries to lock K then it will fail with a timeout as the lock is already held by TX1. The reason for this is the that the lock for a key K is always, deterministically, acquired on the same node of the cluster, regardless of where the transaction originates.

The purpose of simple versioning is to provide a reliable mechanism of write skew checks when using optimistic transactions, REPEATABLE_READ and a clustered cache. Write skew checks are performed at prepare-time to ensure a concurrent transaction hasn’t modified an entry while it was read and potentially updated based on the value read.

When operating in LOCAL mode, write skew checks rely on Java object references to compare differences and this is adequate to provide a reliable write-skew check, however this technique is useless in a cluster and a more reliable form of versioning is necessary to provide reliable write skew checks.

Simple versioning is an implementation of the proposed EntryVersion interface, backed by a long that is incremented each time the entry is updated.

This versioning scheme makes use of vector clocks to provide a network partition resilient form of versioning.

Unlike simple versioning, which is maintained per entry, a vector clock’s node counter is maintained per-node.

This scheme is used to encapsulate an external source of data versioning within Infinispan, such as when using Infinispan with Hibernate which in turn gets it’s data version information directly from a database.

In this scheme, a mechanism to pass in the version becomes necessary, and overloaded versions of put() and putForExternalRead() will be provided in AdvancedCache to take in an external data version. This is then stored on the InvocationContext and applied to the entry at commit time.

Write skew checks cannot and will not be performed in the case of external data versioning.

To deal with deletions of entries, tombstones will be maintained as null entries that have been deleted, so that version information of the deleted entry can be maintained and write skews can still be detected. However this is an expensive thing to do, and as such, is a configuration option, disabled by default. Further, tombstones will follow a strict lifespan and will be cleared from the system after a specific amount of time.

By default versioning will be disabled. This will mean write skew checks when using transactions and REPEATABLE_READ as an isolation level will be unreliable when used in a cluster. Note that this doesn’t affect single-node, LOCAL mode usage.

Or

Since GETs are sent to all data owners in parallel and the first returning result is used, this can lead to data inconsistency when using an asynchronous transport. If an updating thread modifies the primary data owner, but updates are only sent to backup nodes asynchronously, a concurrent read may read a stale value for a short period of time until the asynchronous replication completes.

Note that this is only if the transport is asynchronous. If using a synchronous transport this behavior is not exhibited.

image::[]

Infinispan does not attempt to evenly split the hash space between nodes – by not trying to split it evenly, it means that if a node joins or leaves the grid, there is no need to adjust the ownership of every node, just the neighbours of the joiner/leaver. This has a positive impact on network traffic. However this can mean that some nodes take on substantially larger portions of the hash space than others. This, combined with potential irregularities in the hash functions of keys, can mean the distribution of entries across the grid becomes poor. In order to address the irregularities in the hash of keys, Infinispan uses an advanced hashing function (Murmur Hash 3) by default, as well as using a bit spreader. In order to address the irregularities in the node distribution, Infinispan uses virtual nodes.

First, let’s consider how virtual nodes help conceptually by taking a couple of distribution examples, and armed with that knowledge, look at how Infinispan uses them.

Consider a hash space of 1000 (there are 1000 buckets into which data can be placed). If there were two nodes, it is possible you can have 1 node being used for 1 bucket, and 1 node for 999 buckets (this is the most pessimistic distribution!). If there were 200 nodes, the worst distribution of node would end up being 199 nodes responsible for one bucket each (199 buckets in total), and 1 node being responsible for 801 buckets. If there were 1000 nodes, then each node must be responsible for 1 bucket each. From this we can deduce that as the number of nodes tends to the size of the hash space, that the distribution of buckets to nodes improves.

A guiding principle of Infinispan is that it always uses an algorithm to locate a key in the hash space, never allowing the node on which the entry is stored to be specified manually. This scheme allows any node to know which nodes owns a key, without having to distribute such ownership information. This reduces the overhead of Infinispan, but more importantly improves redundancy as there is no need to replicate the ownership information in case of node failure.

With this in mind, we can see that virtual nodes are an ideal solution to the distribution of nodes problem, as it allows the location of an entry to be determined algorithmically.

Infinispan implements virtual nodes by altering the algorithm for splitting the hash space whenever a node joins or leaves the grid. Rather than allocating a block of the hashspace to the node, it allocates a number of blocks from throughout the hash space.

Often it’s easier to understand the topology changes that virtual nodes introduce through diagrams.

image::[]

image::[]

To use virtual nodes, simply set the number of virtual nodes higher than 1. For example

Alternatively, you can enable virtual nodes programmatically

To prevent repeated remote calls when doing multiple GETs, L1 caching can be enabled. L1 caching places remotely received values in a near cache for a short period of time (configurable) so repeated lookups would not result in remote calls. In the above diagram, if L1 was enabled, a subsequent GET for the same key on Server3 would not result in any remote calls.

image::[]

L1 caching is not free though. Enabling it comes at a cost, and this cost is that every time a key is updated, an invalidation message needs to be multicast to ensure nodes with the entry in L1 invalidates the entry. L1 caching causes the requesting node to cache the retrieved entry locally and listen for changes to the key on the wire. L1-cached entries are given an internal expiry to control memory usage. Enabling L1 will improve performance for repeated reads of non-local keys, but will increase memory consumption to some degree. It offers a nice tradeoff between the "read-mostly" performance of an invalidated data grid with the scalability of a distributed one. Is L1 caching right for you? The correct approach is to benchmark your application with and without L1 enabled and see what works best for your access pattern.

The key affinity service solves the following problem: for a distributed Infinispan cluster one wants to make sure that a value is placed in a certain node. Based on a supplied cluster address identifying the node, the service returns a key that will be hashed to that particular node.

Whenever you add <async> element within <clustering>, you’re telling the underlying JGroups layer in Infinispan to use asynchronous communication. What this means is that JGroups will send any replication/distribution/invalidation request to the wire but will not wait for a reply from the receiver.

This is a configurable boolean property of <async> element that indicates whether the actual call from Infinispan to the JGroups layer is done on a separate thread or not. When set to true, once Infinispan has figured out that a request needs to be sent to another node, it submits it to the async transport executor so that it can talk to the underlying JGroups layer.

With asynchronous marshalling, Infinispan requests can return back to the client quicker compared to when async marshalling is set to false. The downside though is that client requests can be reordered before they have reached the JGroups layer. In other words, JGroups provides ordering guarantees even for async messages but with async marshalling turned on, requests can reach the JGroups in a different order in which they’re called. This can effectively lead to data consistency issues in applications making multiple modifications on the same key/value pair. For example, with async marshalling turned on:

App calls:

Other nodes could receive these operations in this order:

The end result is clearly different which is often not desirable. So, if your application makes multiple modifications on the same key, you should either: turned off asynchronous marshalling, or set <asyncTransportExecutor> element’s maxThreads to 1. The first modification only applies to a particular named cache, whereas the second option affects all named caches in configuration file that are configured with async marshalling. It’s worth noting though that having this type of executor configured with a single thread would defeat its purpose adding unnecessary contention point. It’d be better to simply switch off async marshalling.

On the contrary, if your application only ever makes one modification per key/value pair and there’s no happens-before relationship between them, then async marshalling is a very valid optimization that can increase performance of your application without data consistency risks.

If you have async marshalling turned on and see exceptions related to java.util.concurrent.RejectedExecutionException, you should also consider switching off async marshalling.

The aim of the replication queue is to batch the individual cache operations and send them as one, as opposed to sending each cache operation individually. As a result, replication queue enabled configurations perform generally better compared to those that have it switched off because less RPC messages are sent, fewer envelopes are used…etc. The only real trade off to the replication queue is that the queue is flushed periodically (based on time or queue size) and hence it might take longer for the replication/distribution/invalidation to be realised across the cluster. When replication queue is turned off, data is placed directly on the wire and hence it takes less for data to arrive to other nodes.

Finally, the asynchronous API can be used to emulate non-blocking APIs, whereby calls are handed over to a different thread and asynchronous API calls return to the client immediately. Similar to async marshalling, using this API can lead to reordering, so you should avoid calling modifying asynchronous methods on the same keys.

Regardless of the asynchronous option used, the return values of cache operations are reliable. If talking about return values of cache operations that return previous value, the correctness of these returns are guaranteed as well regardless of the clustering mode. With replication, the previous value is already available locally, and with distribution, regardless of whether it’s asynchronous or synchronous, Infinispan sends a synchronous request to get the previous value if not present locally. If on the other hand the asynchronous API is used, client code needs to get hold of the NotifiyngFuture returned by the async operation in order to be able to query the previous value.

Store as binary enables data to be stored in its serialized form. This can be useful to achieve lazy deserialization, which is the mechanism by which Infinispan by which serialization and deserialization of objects is deferred till the point in time in which they are used and needed. This typically means that any deserialization happens using the thread context class loader of the invocation that requires deserialization, and is an effective mechanism to provide classloader isolation. By default lazy deserialization is disabled but if you want to enable it, you can do it like this:

Sometimes it might happen that the Infinispan marshalling layer, and in particular JBoss Marshalling, might have issues marshalling/unmarshalling some user object. In Infinispan 4.0, marshalling exceptions will contain further information on the objects that were being marshalled. Example:

The way the "in object" messages are read is the same in which stacktraces are read. The highest "in object" being the most inner one and the lowest "in object" message being the most outer one. So, the above example indicates that a java.lang.Object instance contained in an instance of org.infinispan.commands.write.PutKeyValueCommand could not be serialized because java.lang.Object@b40ec4 is not serializable.

This is not all though! If you enable DEBUG or TRACE logging levels, marshalling exceptions will contain show the toString() representations of objects in the stacktrace. For example:

With regards to unmarshalling exceptions, showing such level of information it’s a lot more complicated but where possible. Infinispan will provide class type information. For example:

In this example, an IOException was thrown when trying to unmarshall a instance of the inner class org.infinispan.marshall.VersionAwareMarshallerTest$1. In similar fashion to marshalling exceptions, when DEBUG or TRACE logging levels are enabled, classloader information of the class type is provided. For example:

Finding the root cause of marshalling/unmarshalling exceptions can sometimes be really daunting but we hope that the above improvements would help get to the bottom of those in a more quicker and efficient manner.

By default you can inject the default Infinispan cache. Let’s look at the following example:

If you want to use a specific cache rather than the default one, you just have to provide your own cache configuration and cache qualifier. For example, if you want to use a custom cache for the GreetingService you should write your own qualifier (such as GreetingCache) and define its configuration:

To use this cache in the GreetingService add the @GeetingCache qualifier on your cache injection point. Simple!

You can override the default cache configuration used by the default embedded cache manager. For that, you just have to create one Configuration producer with the @Default qualifier as illustrated in the following snippet:

It’s also possible to override the default embedded cache manager used. The new default cache manager produced must have the @Default qualifier and the scope @ApplicationScoped .

To use Infinispan in a clustered mode you have to configure the transport with the GlobalConfiguration. To achieve that override the default cache manager as explained in the previous section. Look at the following snippet:

With the CDI integration it’s also possible to use a remote cache. For example you can inject the default RemoteCache as illustrated in the following snippet:

If you want to use another cache, for example the greeting-cache, add the @Remote qualifier on the cache injection point which contains the cache name.

Adding the @Remote cache qualifier on each injection point might be error prone. That’s why the remote cache integration provides another way to achieve the same goal. For that you have to create your own qualifier annotated with @Remote :

To use this cache in the GreetingService add the qualifier @RemoteGreetingCache qualifier on your cache injection.

Like the embedded cache integration, the remote cache integration comes with a default remote cache manager producer. This default remote cache manager can be overridden as illustrated in the following snippet:

JMX reporting can be enabled at 2 different levels:

The CacheManager is the entity that governs all the cache instances that have been created from it. Details on the information exposed at the CacheManager level can be found below. For the moment, let’s just focus on how to enable the CacheManager to report management data via JMX.

At this level, you will receive management information generated by individual cache instances. Details on the information exposed at the Cache level are explained below. For the moment, let’s just focus on how to enable the Cache to report management data via JMX:

Once you have enabled JMX reporting at either the CacheManager or Cache level, you should be able to connect to VM(s) where Infinispan is running using a standard JMX GUI such as JConsole or VisualVM, and you should find the following MBeans:

For example, the cache store JMX component MBean for a default cache configured with synchronous distribution would have the following name: org.infinispan:type=Cache,name=" _ defaultcache(dist_sync)", manager="DefaultCacheManager",component=CacheStore

Please note that cache and cache manager names are quoted to protect against illegal characters being used in these user-defined names.

There can be situations where several CacheManager instances are created in a single VM, or Cache names belonging to different CacheManagers under the same VM clash.

Using different JMX domains for multi cache manager environments should be last resort. Instead, it’s possible to name a cache manager in such way that it can easily be identified and used by monitoring tools such as RHQ. For example:

Using either of these options should result on the CacheManager MBean name being: org.infinispan:type=CacheManager,name="Hibernate2LC"

For the time being, you can still set your own jmxDomain if you need to and we also allow duplicate domains, or rather duplicate JMX names, but these should be limited to very special cases where different cache managers within the same JVM are named equally.

Let’s discuss where Infinispan registers all these MBeans. By default, Infinispan registers them in the standard JVM MBeanServer plattform . However, users might want to register these MBeans in a different MBeanServer instance. For example, an application server might work with a different MBeanServer instance to the default plattform one. In such cases, users should implement the MBeanServerLookup interface provided by Infinispan so that the getMBeanServer() method returns the MBeanServer under which Infinispan should register the management MBeans. You can find an example in the default PlatformMBeanServerLookup class used by Infinispan. So, once you have your implementation ready, simply configure Infinispan with the fully qualified name of this class. For example:

There has been a couple of noticeable additions in Infinispan 5.0 in terms of MBean exposed:

This section focuses on the lessons learned while developing the Infinispan RHQ plugin that are likely to be useful to anyone using RHQ.

HTTP GET and HEAD are used to retrieve data from entries.

Data can be removed at the cache key/element level, or via a whole cache name using the HTTP delete method.

Indexing must be enabled in the configuration (as explained in XML Configuration or Programmatic configuration ). This will trigger automatic indexing of objects stored in the cache; there are several different ways to specify how these objects need to be indexed explained in the following paragraphs. To run queries you use the SearchManager which exposes all necessary methods to get started.

Using @DocumentId to mark a field as identifier does not apply to Infinispan values; in Infinispan Query the identifier for all @Indexed objects is the key used to store the value. You can still customize how the key is indexed using a combination of @Transformable , @ProvidedId , custom types and custom FieldBridge implementations.

The key for each value needs to be indexed as well, and the key instance must be transformed in a String. Infinispan includes some default transformation routines to encode common primitives, but to use a custom key you must provide an implementation of org.infinispan.query.Transformer .

To enable indexing via XML, you need to add the <indexing … /> element to your cache configuration, and optionally pass additional properties to the embedded Hibernate Search engine:

In this example the index is stored in memory, so when this nodes is shutdown the index is lost: good for a quick demo, but in real world cases you’ll want to use the default (store on filesystem) or store the index in Infinispan as well. For the complete reference of properties to define, refer to the Hibernate Search documentation .

Infinispan Query isn’t aware of where you store the indexes, it just passes the configuration of which Lucene Directory implementation you want to use to the Hibernate Search engine. There are several Lucene Directory implementations bundled, and you can plug your own or add third party implementations: the Directory is the IO API for Lucene to store the indexes.

The most common Lucene Directory implementations used with Infinispan Query are:

Of course having a shared index vs. an independent index on each node directly affects behaviour of the Query module; some combinations might not make much sense.

In the following example we start Infinispan programmatically, avoiding XML configuration files, and also map an object Author which is to be stored in the grid and made searchable on two properties but without annotating the class.

In local mode, you may use any Lucene Directory implementation. Also the option indexLocalOnly isn’t meaningful.

In replication mode, each node can have it’s own local copy of the index. So indexes can either be stored locally on each node (RAMDirectory, FSDirectory, etc) but you need to set indexLocalOnly to false , so that each node will apply needed updates it receives from other nodes in addition to the updates started locally. Any Directory implementation can be used, but you have to make sure that when a new node is started it receives an up to date copy of the index; typically rsync is well suited for this task, but being an external operation you might end up with a slightly out-of-sync index, especially if updates are very frequent.

Alternately, if you use some form of shared storage for indexes (see Sharing the Index ), you then have to set indexLocalOnly to true so that each node will apply only the changes originated locally; in this case there’s no risk in having an out-of-sync index, but to avoid write contention on the index you should make sure that a single node is "in charge" of updating the index. Again, the Hibernate Search reference documentation describes means to use a JMS queue or JGroups to send indexing tasks to a master node.

The diagram below shows a replicated deployment, in which each node has a local index.

For these 2 cache modes, you need to use a shared index and set indexLocalOnly to true.

The diagram below shows a deployment with a shared index. Note that while not mandatory, a shared index can be used for replicated (vs. distributed) caches as well.

image::[]

Indexing or searching of elements under INVALIDATION mode is not supported.

Put data under a specified key.

Get the data stored under the specified key.

Delete the entry with the specified key.

Clear the cache

Verify if the entry exists in the cache

Get version of the entry. Version format differs between protocols:

REST client implements this operation efficiently by executing HEAD method.

Print help about an operation

Change internal state/config of the client. This operation has only client-side effect.

Process cache commands from the specified batch file. The commands will be processed line by line.

When exchanging data via REST interface, the values are interpreted by any client as sequence of bytes. The meaning is given to this byte-sequence by using MIME type specified via "Content-Type" HTTP header. No special interoperability measures are needed here.

If we want to read in ispncon the entries that were put with Hot Rod Java client, we need to use a special option hotrod.use_river_string_keys = True. This will cause the string keys to be encoded the same way the Java client does it.

Using hotrod.use_river_string_keys = True we’re able to access the data that has been written by the java client, but we still see the raw binary values. To be able to see a value that has been put by Hot Rod java client in a readable form and vice versa - to be able to see in Hot Rod Java client what we’ve put via ispncon we need to use a codec . Currently there are two types of codecs: RiverString and RiverByteArray

Codecs can be used either by specifying a default_codec option in the ~/.ispncon config file (in section ispncon) or by specifying a codec on each put resp get using -e (--encode) resp -d (--decode) options .

Tested with spymemcached 2.7.

Values stored by ispncon is interpreted as an UTF-8 string, meaning if we store data using ispncon, it will be interpreted by the Java client as a String by calling new java.lang.String(bytes, "UTF-8").

This also works in reverse: values stored in Java as java.lang.String will be returned as UTF-8 bytes in ispncon

1. First of all, to enable JPA/Hibernate second level cache with query result caching enabled, add either of the following:

2. Now, configure the Infinispan cache region factory using one of the two options below:

This is all the configuration you need to have JPA/Hibernate use Infinispan as cache provider with the default settings. You will still need to define which entities and queries need to be cached as defined in the Hibernate reference documentation, but that configuration aspect is not peculiar to Infinispan. This default configuration should suit the majority of use cases but sometimes, further configuration is required and to help with such situations, please check the following section where more advanced settings are discussed.

The aim of this section is to explain the default settings for each of the different global data type (entity, collection, query and timestamps) caches, why these were chosen and what are the available alternatives.

It is highly recommended that Hibernate is configured with JTA transactions so that both Hibernate and Infinispan cooperate within the same transaction and the interaction works as expected.

Otherwise, if Hibernate is configured for example with JDBC transactions, Hibernate will create a Transaction instance via java.sql.Connection and Infinispan will create a transaction via whatever TransactionManager returned by hibernate.transaction.manager_lookup_class. If hibernate.transaction.manager_lookup_class has not been populated, it will default on the dummy transaction manager. So, any work on the 2nd level cache will be done under a different transaction to the one used to commit the stuff to the database via Hibernate. In other words, your operations on the database and the 2LC are not treated as a single unit. Risks here include failures to update the 2LC leaving it with stale data while the database committed data correctly. It has also been observed that under some circumstances where JTA was not used, commit/rollbacks are not propagated to Infinispan.

To sum up, if you configure Hibernate with Infinispan, apply the following changes to your configuration file:

1. Unless your application uses JPA, you need to select the correct Hibernate transaction factory via the property hibernate.transaction.factory_class :

The reason why JPA does not require a transaction factory class to be set up is because the entity manager already sets it to a variant of CMTTransactionFactory.

2. Select the correct Hibernate transaction manager lookup:

For example if you were running with the JBoss Application Server you would set: